Hardware assisted virtual switch

ABSTRACT

There is disclosed an example of a computing apparatus for providing a hardware-assisted virtual switch on a host, including: a hardware virtual switch (vSwitch) circuit; and a hardware virtual host (vHost) circuit, the vHost circuit having an interface driver specific to the hardware vSwitch and configured to provide a vHost data plane to: provide a plurality of hardware queues to communicatively couple the hardware vSwitch to a guest virtual function (VF); and present to a virtual network driver of the guest VF an interface that is backward compatible with a software network interface.

FIELD OF THE SPECIFICATION

This disclosure relates in general to the field of cloud computing, andmore particularly, though not exclusively, to a system and method for ahardware assisted virtual switch.

BACKGROUND

In some modern data centers, the function of a device or appliance maynot be tied to a specific, fixed hardware configuration. Rather,processing, memory, storage, and accelerator functions may in some casesbe aggregated from different locations to form a virtual “compositenode.” A contemporary network may include a data center hosting a largenumber of generic hardware server devices, contained in a server rackfor example, and controlled by a hypervisor. Each hardware device mayrun one or more instances of a virtual device, such as a workload serveror virtual desktop.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is best understood from the following detaileddescription when read with the accompanying FIGURES. It is emphasizedthat, in accordance with the standard practice in the industry, variousfeatures are not necessarily drawn to scale, and are used forillustration purposes only. Where a scale is shown, explicitly orimplicitly, it provides only one illustrative example. In otherembodiments, the dimensions of the various features may be arbitrarilyincreased or reduced for clarity of discussion.

FIG. 1 is a block diagram of selected components of a data center withnetwork connectivity, according to one or more examples of the presentspecification.

FIG. 2 is a block diagram of selected components of an end-usercomputing device, according to one or more examples of the presentspecification.

FIG. 3 is a block diagram of a software-defined network (SDN), accordingto one or more examples of the present specification.

FIG. 4 is a block diagram of a network function virtualization (NFV)architecture, according to one or more examples of the presentspecification.

FIG. 5 is a block diagram of a virtualized network functioninfrastructure (VNFI), according to one or more examples of the presentspecification.

FIG. 6 illustrates a VNFI, according to one or more examples of thepresent specification.

FIG. 7 is a block diagram of a virtual machine (VM) migration, accordingto one or more examples of the present specification.

FIG. 8 is a signal flow diagram illustrating a guest transmit operation,according to one or more examples of the present specification.

FIG. 9 is a signal flow diagram illustrating a live migration event,according to one or more examples of the present specification.

FIG. 10 illustrates a VNFI, according to one or more examples of thepresent specification.

EMBODIMENTS OF THE DISCLOSURE

The following disclosure provides many different embodiments, orexamples, for implementing different features of the present disclosure.Specific examples of components and arrangements are described below tosimplify the present disclosure. These are, of course, merely examplesand are not intended to be limiting. Further, the present disclosure mayrepeat reference numerals and/or letters in the various examples. Thisrepetition is for the purpose of simplicity and clarity and does not initself dictate a relationship between the various embodiments and/orconfigurations discussed. Different embodiments may have differentadvantages, and no particular advantage is necessarily required of anyembodiment.

A contemporary computing platform, such as a hardware platform providedby Intel® or similar, may include a capability for monitoring deviceperformance and making decisions about resource provisioning. Forexample, in a large data center such as may be provided by a cloudservice provider (CSP), the hardware platform may include rackmountedservers with compute resources such as processors, memory, storagepools, accelerators, and other similar resources. As used herein, “cloudcomputing” includes network-connected computing resources and technologythat enables ubiquitous (often worldwide) access to data, resources,and/or technology. Cloud resources are generally characterized by greatflexibility to dynamically assign resources according to currentworkloads and needs. This can be accomplished, for example, viavirtualization, wherein resources such as hardware, storage, andnetworks are provided to a virtual machine (VM) via a softwareabstraction layer, and/or containerization, wherein instances of networkfunctions are provided in “containers” that are separated from oneanother, but that share underlying operating system, memory, and driverresources.

In the modern data center, and particularly in the context of a CSP,data paths are often highly optimized. Because a data center may includehardware devices that are separate from one another and that areconnected by a high-speed fabric, it is often necessary to ensure thatthe fabric can operate at speeds similar to or faster than localinterconnects on an older monolithic server device wherein all thehardware may be collocated.

For example, a CSP may provide a number of virtual machines to itscustomers, and its business model may include selling central processingunit (CPU) cores, CPU core time, memory, network bandwidth, storage, anddefined service level agreements of bandwidth and interoperabilitybetween these to its end users. The client can run applications on thesehardware resources to provide their own network services, as though theywere maintaining their own individual server ecosystems. Theseapplications may be chained together in a service chain running onvarious virtual machines as part of a “tenant” of a server cloud.

The various VMs that the customer runs may all be interconnected via avirtual switch (vSwitch), which in existing software-only systems mayinclude an application running on the host. The application itself canuse anywhere from 1 to 8 cores depending on the workload assigned to it,meaning that those 1 to 8 cores per physical server can no longer besold to end users. They are simply consumed as overhead.

To reduce the overhead in compute cores consumed by vSwitch, it ispossible to provide the vSwitch in hardware. For example, an existinghardware driver for a network interface card (NIC) that would be used ina physical switch can be integrated into a rackmount server. However,rather than physically coupling to physical Ethernet switches, the chipis simply programmed to provide switching between virtual ports thatexist on VMs on the server. While the use of a hardware-only vSwitch canindeed eliminate the need to consume one or more cores to provide asoftware vSwitch, this often comes at the cost of flexibility. A virtualmachine running on a host of the hardware-only vSwitch may need to loadan appropriate driver for that hardware, similar to how the operatingsystem kernel on a standalone server using a hardware network card wouldneed a driver for that network card. This inhibits the ability to livemigrate VMs between dissimilar platforms. For example, a VM running on asystem with such a hardware-only vSwitch could be live migrated to anidentical hardware platform having an identical hardware vSwitch, butmigrating that VM to a hardware platform having a differenthardware-only vSwitch would require (at a minimum) reloading the driversand updating the configuration, possibly even requiring a reboot of theserver appliance. Migrating a VM from a hardware platform providing ahardware-only vSwitch to a platform that provides only a softwarevSwitch could be even more difficult and require even more changes tothe configuration.

The present specification describes a system and method for providing ahardware assisted vSwitch that realizes the benefits of a hardwarevSwitch by abstracting certain operations into a vHost net driver thatcan be backward compatible with software-only vSwitches, and can alsotake advantage of various hardware vSwitches regardless of theirspecific identities.

For example, certain existing solutions connect the vSwitch hardwaredirectly into a guest VM or container using single root input/outputvirtualization (SR-IOV). The guest is given an SR-IOV virtual function(VF) using “virtual function (VF) passthrough.” As described above,SR-IOV requires a hardware specific driver in the guest to be run on thepassthrough device, and is limited in its ability to forward packetsbetween guests to a layer 2 semantic, based on only media access control(MAC) addresses. VF passthrough breaks the ability for live migration.

In some cases, a solution can be provided utilizing SR-IOV and ahardware vSwitch to replace the layer 2 forwarding of SR-IOV. Thisapproach may be referred to as “vSwitch-controlled SR-IOV.” In thismode, vSwitch forwarding is applied to traffic, but a VF passthrough isstill required to connect VM guest applications to the network. Thisapproach still requires a VF to be passed through, for example, in QuickEmulator (QEMU), and thus is also not compatible with live migration.

Thus, a distinction between SR-IOV and the disclosure of the presentspecification is that instead of passing an entire VF into the guest,the hardware stays associated with the underlying host, and replaces thevirtual descriptor rings (vrings) that may otherwise have been emulatedin software in vHost. This offloads the fast data path connecting the VMto the hardware offloaded vSwitch (in other words, the work that oftenconsumes large numbers of CPU cycles), but leaves the control plane forthis connectivity in software.

Stated differently, existing vSwitches include a virtual function thatpresents a standardized network interface to the guest (virtio).Software in the guest and software in the host communicate as thoughthey were two ends of the same wire. A Linux host may include a vHostacceleration layer that can consume one or more cores just to optimallyhandle the data paths of the virtio backend. The vHost thread may, forexample, get guest memory address space from the hypervisor, and watchthis address space for doorbells, to see when traffic is ready to besent, and then send and receive traffic. The thread may also ring aninterrupt when the operation completes. This provides a highly-optimizedsoftware vSwitch, at the cost of one or more cores.

The present specification provides a hardware-assisted vSwitch in whichvHost functions may be offloaded to a hardware device on the hostdevice. The host hypervisor may then not need to allocate a vHostthread. Instead, the hardware vHost receives the address mapping, and itprograms those mappings into its hardware using, for example, anexisting IOMMU. The hardware vHost maps the guest private address (GPA)to the host private address (HPA), and programs this into the IOMMU.

Note that the hardware vHost may be specific to the network softwarebackend on the host, and is thus able to present an abstracted interfaceto all guests, eliminating the need for guests to provide their ownkernel-level drivers for the networking hardware. Rather, the guest cansimply use the standard emulated driver (virtio). This means that a hostdevice with a hardware vSwitch appears identical to a host with asoftware vSwitch, as they use the identical virtio emulated driver.Furthermore, because the hardware vHost is specific to the vSwitchhardware IP, VMs can be live migrated between hosts with differenthardware IP blocks, or between hosts with software vSwitches andhardware vSwitches. The VM itself can remain agnostic of what kind ofvSwitch the host is running.

The hardware vHost can read data directly between the host and guest.Instead of a software thread reading and writing data (software vHost),the hardware vHost reads and writes data, and also handles doorbells andinterrupts.

Some older versions of the virtio-net driver in the guest may not befully compatible with a hardware vHost. For example, a relay mechanismmay support the ability to convert from a pure software ‘notify’ that anew frame is ready to send. Some cycles may still be consumed inoperating the relay, but the cycles consumed in this operation are muchfewer than the cycles that would be consumed in a software-only vSwitch.Furthermore, this system does not break the hypervisor construct, thusmaintaining backward compatibility. CPU overhead is significantlyreduced, while functions available in software-only vSwitches such aslive migration are preserved. Advantageously, this system provides ahardware vSwitch, but eliminates the need of having the hardwareimplement the control plane.

Embodiments of the present specification use virtio-net as the hostinterface. Virtio-net is an existing open source standardparavirtualized driver. Thus, existing hardware platforms have alreadybeen designed to conform to the virtio-net driver’s existing behavior.Advantageously, existing virtio-net drivers in the guest VM can be usedunmodified. The capabilities supported by the hardware may be dependenton the version of virtio-net driver provided in the guest.

Embodiments of this specification also provide “vHost offload.” vHostoffload does not pass through a VF device into the guest, but ratheremulates the behavior of a software vHost, using the VFs that remain inthe host. The software vHost instantiates a set of emulated cues thatcan be passed to a VM at instantiation to implement the back end of thevirtio-net driver. vHost offload may replace these emulated cues withhardware cues from a virtio-net VF. Thus, the vHost control plane (e.g.,providing the ability to negotiate and live migrate the system) stays inthe software.

By way of nonlimiting example, vHost offload may support the followingfeatures:

-   Virtio-net Notify Kick Relay. When a virtio-net driver is used in    the guest, it may choose to notify the backend of new events by    using a fast event path (eventfd) kick via programmed input/output    (PIO). The kick is received on the backend side just as in an    emulated vHost, and in this case the kick may be translated into a    hardware write on the vHost. This enables virtio-net notifies with    hardware based cues.-   Virtio-net Interrupt Request (IRQ) Relay in Contemporary Linux-Based    Systems. For both versions of virtio-net, the IRQ is filtered    through QEMU just as the notify kick. This IRQ relay translates the    hardware interrupt in the host into a software fast interrupt    request path (IRQFD) in the guest.-   Guest Data Address Remapping. Virtio-net guests may write into their    descriptors addresses pointing to the data in their own guest    physical address (GPA) space. In a software vHost, the GPA is    translated into a host physical address (HPA) using a mapping that    comes from QEMU. However, in the vHost offload approach, the mapping    is populated into the input/output memory management unit (IOMMU) to    allow hardware to address the data by GPA, finding it in the right    location at the HPA.-   Live Migration Handling. When a live migration is triggered, the    program state of which pages are in use may be transferred to QEMU    so that it can reset the mapping and the destination host. All    descriptor ring states held in the device (i.e., not in host memory)    should also be transmitted. Thus, all the active pages are dirtied    and swapped out of the cache, and the full device state is carried    in alongside the VM, so that the current state can be transferred    during migration. On the destination side, the new hardware platform    may have the capability to load the current running state of the    vrings and warm start from where the source rings left off.

vHost offload provides a relay type of approach, with some optimizationsprovided because the descriptor queue structure is the same on bothsides of the relay. Interrupts may be relayed as there may not beanother mechanism to pass interrupts directly into a virtualized guestwithout VF passthrough. Optimizations provided by vHost offload include,by way of nonlimiting example:

-   Data Zero Copy. For RX and TX, the IOMMU is used to map GPA into    HPA, so that the hardware can access guest pages and read or write    data directly into these pages. In some embodiments, it may be    necessary for the guest’s memory to be pinned in the host memory    (either the entire memory space, or at a minimum the space used for    virtio-net). This may be particularly true for guests that run Data    Plane Developer Kit (DPDK) with HugePages, while for regular kernel    guests, this may be a novel method of instantiating guests on the    host.-   Single Descriptor Table, Avail, and Used Rings. A relay may support    the ability to convert from two different descriptor ring    structures, although this may involve performing the conversion in    software, thus consuming CPU cycles. Using the present approach, the    descriptor table and rings may be accessed directly between the    guest and the underlying hardware.-   Guest Direct Hardware Kick. In virtio-net, the guest may directly    kick the hardware by writing the “avail” index via the MMIO.-   Interrupt Remapping. By way of instructions provided in software, an    interrupt remapping table may be provided in the IOMMU, enabling    vHost offload hardware to pass interrupts into a virtualized guest.    By leveraging the interrupt remapping feature of IOMMU (e.g., posted    interrupt), a hypervisor does not intervene in the relay.

A system and method for a hardware assisted virtual switch will now bedescribed with more particular reference to the attached FIGURES. Itshould be noted that throughout the FIGURES, certain reference numeralsmay be repeated to indicate that a particular device or block is whollyor substantially consistent across the FIGURES. This is not, however,intended to imply any particular relationship between the variousembodiments disclosed. In certain examples, a genus of elements may bereferred to by a particular reference numeral (“widget 10”), whileindividual species or examples of the genus may be referred to by ahyphenated numeral (“first specific widget 10-1” and “second specificwidget 10-2”).

FIG. 1 is a block diagram of selected components of a data center withconnectivity to network 100 of a cloud service provider (CSP) 102,according to one or more examples of the present specification. CSP 102may be, by way of nonlimiting example, a traditional enterprise datacenter, an enterprise “private cloud,” or a “public cloud,” providingservices such as infrastructure as a service (IaaS), platform as aservice (PaaS), or software as a service (SaaS).

CSP 102 may provision some number of workload clusters 118, which may beclusters of individual servers, blade servers, rackmount servers, or anyother suitable server topology. In this illustrative example, twoworkload clusters, 118-1 and 118-2 are shown, each providing rackmountservers 146 in a chassis 148.

In this illustration, workload clusters 118 are shown as modularworkload clusters conforming to the rack unit (“U”) standard, in which astandard rack, 19 inches wide, may be built to accommodate 42 units (42U), each 1.75 inches high and approximately 36 inches deep. In thiscase, compute resources such as processors, memory, storage,accelerators, and switches may fit into some multiple of rack units fromone to 42.

Each server 146 may host a standalone operating system and provide aserver function, or servers may be virtualized, in which case they maybe under the control of a virtual machine manager (VMM), hypervisor,and/or orchestrator, and may host one or more virtual machines, virtualservers, or virtual appliances. These server racks may be collocated ina single data center, or may be located in different geographic datacenters. Depending on the contractual agreements, some servers 146 maybe specifically dedicated to certain enterprise clients or tenants,while others may be shared.

The various devices in a data center may be connected to each other viaa switching fabric 170, which may include one or more high speed routingand/or switching devices. Switching fabric 170 may provide both“north-south” traffic (e.g., traffic to and from the wide area network(WAN), such as the internet), and “east-west” traffic (e.g., trafficacross the data center). Historically, north-south traffic accounted forthe bulk of network traffic, but as web services become more complex anddistributed, the volume of east-west traffic has risen. In many datacenters, east-west traffic now accounts for the majority of traffic.

Furthermore, as the capability of each server 146 increases, trafficvolume may further increase. For example, each server 146 may providemultiple processor slots, with each slot accommodating a processorhaving four to eight cores, along with sufficient memory for the cores.Thus, each server may host a number of VMs, each generating its owntraffic.

To accommodate the large volume of traffic in a data center, a highlycapable switching fabric 170 may be provided. Switching fabric 170 isillustrated in this example as a “flat” network, wherein each server 146may have a direct connection to a top-of-rack (ToR) switch 120 (e.g., a“star” configuration), and each ToR switch 120 may couple to a coreswitch 130. This two-tier flat network architecture is shown only as anillustrative example. In other examples, other architectures may beused, such as three-tier star or leaf-spine (also called “fat tree”topologies) based on the “Clos” architecture, hub-and-spoke topologies,mesh topologies, ring topologies, or 3-D mesh topologies, by way ofnonlimiting example.

The fabric itself may be provided by any suitable interconnect. Forexample, each server 146 may include an Intel® Host Fabric Interface(HFI), a network interface card (NIC), or other host interface. The hostinterface itself may couple to one or more processors via aninterconnect or bus, such as PCI, PCIe, or similar, and in some cases,this interconnect bus may be considered to be part of fabric 170.

The interconnect technology may be provided by a single interconnect ora hybrid interconnect, such as where PCIe provides on-chipcommunication, 1Gb or 10Gb copper Ethernet provides relatively shortconnections to a ToR switch 120, and optical cabling provides relativelylonger connections to core switch 130. Interconnect technologiesinclude, by way of nonlimiting example, Intel® Omni-Path™, TrueScale™,Ultra Path Interconnect (UPI) (formerly called QPI or KTI),FibreChannel, Ethernet, FibreChannel over Ethernet (FCoE), InfiniBand,PCI, PCIe, or fiber optics, to name just a few. Some of these will bemore suitable for certain deployments or functions than others, andselecting an appropriate fabric for the instant application is anexercise of ordinary skill.

Note however that while high-end fabrics such as Omni-Path™ are providedherein by way of illustration, more generally, fabric 170 may be anysuitable interconnect or bus for the particular application. This could,in some cases, include legacy interconnects like local area networks(LANs), token ring networks, synchronous optical networks (SONET),asynchronous transfer mode (ATM) networks, wireless networks such asWiFi and Bluetooth, “plain old telephone system” (POTS) interconnects,or similar. It is also expressly anticipated that in the future, newnetwork technologies will arise to supplement or replace some of thoselisted here, and any such future network topologies and technologies canbe or form a part of fabric 170.

In certain embodiments, fabric 170 may provide communication services onvarious “layers,” as originally outlined in the OSI seven-layer networkmodel. In contemporary practice, the OSI model is not followed strictly.In general terms, layers 1 and 2 are often called the “Ethernet” layer(though in large data centers, Ethernet has often been supplanted bynewer technologies). Layers 3 and 4 are often referred to as thetransmission control protocol/internet protocol (TCP/IP) layer (whichmay be further subdivided into TCP and IP layers). Layers 5 - 7 may bereferred to as the “application layer.” These layer definitions aredisclosed as a useful framework, but are intended to be nonlimiting.

FIG. 2 is a block diagram of a data center 200 according to one or moreexamples of the present specification. Data center 200 may be, invarious embodiments, the same as network 100 of FIG. 1 , or may be adifferent data center. Additional views are provided in FIG. 2 toillustrate different aspects of data center 200.

In this example, a fabric 270 is provided to interconnect variousaspects of data center 200. Fabric 270 may be the same as fabric 170 ofFIG. 1 , or may be a different fabric. As above, fabric 270 may beprovided by any suitable interconnect technology. In this example,Intel® Omni-Path™ is used as an illustrative and nonlimiting example.

As illustrated, data center 200 includes a number of logic elementsforming a plurality of nodes. It should be understood that each node maybe provided by a physical server, a group of servers, or other hardware.Each server may be running one or more virtual machines as appropriateto its application.

Node 0 208 is a processing node including a processor socket 0 andprocessor socket 1. The processors may be, for example, Intel® Xeon™processors with a plurality of cores, such as 4 or 8 cores. Node 0 208may be configured to provide network or workload functions, such as byhosting a plurality of virtual machines or virtual appliances.

Onboard communication between processor socket 0 and processor socket 1may be provided by an onboard uplink 278. This may provide a very highspeed, short-length interconnect between the two processor sockets, sothat virtual machines running on node 0 208 can communicate with oneanother at very high speeds. To facilitate this communication, a virtualswitch (vSwitch) may be provisioned on node 0 208, which may beconsidered to be part of fabric 270.

Node 0 208 connects to fabric 270 via an HFI 272. HFI 272 may connect toan Intel® Omni-Path™ fabric. In some examples, communication with fabric270 may be tunneled, such as by providing UPI tunneling over Omni-Path™.

Because data center 200 may provide many functions in a distributedfashion that in previous generations were provided onboard, a highlycapable HFI 272 may be provided. HFI 272 may operate at speeds ofmultiple gigabits per second, and in some cases may be tightly coupledwith node 0 208. For example, in some embodiments, the logic for HFI 272is integrated directly with the processors on a system-on-a-chip. Thisprovides very high speed communication between HFI 272 and the processorsockets, without the need for intermediary bus devices, which mayintroduce additional latency into the fabric. However, this is not toimply that embodiments where HFI 272 is provided over a traditional busare to be excluded. Rather, it is expressly anticipated that in someexamples, HFI 272 may be provided on a bus, such as a PCIe bus, which isa serialized version of PCI that provides higher speeds than traditionalPCI. Throughout data center 200, various nodes may provide differenttypes of HFIs 272, such as onboard HFIs and plug-in HFIs. It should alsobe noted that certain blocks in a system on a chip may be provided asintellectual property (IP) blocks that can be “dropped” into anintegrated circuit as a modular unit. Thus, HFI 272 may in some cases bederived from such an IP block.

Note that in “the network is the device” fashion, node 0 208 may providelimited or no onboard memory or storage. Rather, node 0 208 may relyprimarily on distributed services, such as a memory server and anetworked storage server. Onboard, node 0 208 may provide onlysufficient memory and storage to bootstrap the device and get itcommunicating with fabric 270. This kind of distributed architecture ispossible because of the very high speeds of contemporary data centers,and may be advantageous because there is no need to over-provisionresources for each node. Rather, a large pool of high-speed orspecialized memory may be dynamically provisioned between a number ofnodes, so that each node has access to a large pool of resources, butthose resources do not sit idle when that particular node does not needthem.

In this example, a node 1 memory server 204 and a node 2 storage server210 provide the operational memory and storage capabilities of node 0208. For example, memory server node 1 204 may provide remote directmemory access (RDMA), whereby node 0 208 may access memory resources onnode 1 204 via fabric 270 in a DMA fashion, similar to how it wouldaccess its own onboard memory. The memory provided by memory server 204may be traditional memory, such as double data rate type 3 (DDR3)dynamic random access memory (DRAM), which is volatile, or may be a moreexotic type of memory, such as a persistent fast memory (PFM) likeIntel® 3D Crosspoint™ (3DXP), which operates at DRAM-like speeds, but isnonvolatile.

Similarly, rather than providing an onboard hard disk for node 0 208, astorage server node 2 210 may be provided. Storage server 210 mayprovide a networked bunch of disks (NBOD), PFM, redundant array ofindependent disks (RAID), redundant array of independent nodes (RAIN),network attached storage (NAS), optical storage, tape drives, or othernonvolatile memory solutions.

Thus, in performing its designated function, node 0 208 may accessmemory from memory server 204 and store results on storage provided bystorage server 210. Each of these devices couples to fabric 270 via aHFI 272, which provides fast communication that makes these technologiespossible.

By way of further illustration, node 3 206 is also depicted. Node 3 206also includes a HFI 272, along with two processor sockets internallyconnected by an uplink. However, unlike node 0 208, node 3 206 includesits own onboard memory 222 and storage 250. Thus, node 3 206 may beconfigured to perform its functions primarily onboard, and may not berequired to rely upon memory server 204 and storage server 210. However,in appropriate circumstances, node 3 206 may supplement its own onboardmemory 222 and storage 250 with distributed resources similar to node 0208.

The basic building block of the various components disclosed herein maybe referred to as “logic elements.” Logic elements may include hardware(including, for example, a software-programmable processor, an ASIC, oran FPGA), external hardware (digital, analog, or mixed-signal),software, reciprocating software, services, drivers, interfaces,components, modules, algorithms, sensors, components, firmware,microcode, programmable logic, or objects that can coordinate to achievea logical operation. Furthermore, some logic elements are provided by atangible, non-transitory computer-readable medium having stored thereonexecutable instructions for instructing a processor to perform a certaintask. Such a non-transitory medium could include, for example, a harddisk, solid state memory or disk, read-only memory (ROM), persistentfast memory (PFM) (e.g., Intel® 3D Crosspoint™), external storage,redundant array of independent disks (RAID), redundant array ofindependent nodes (RAIN), network-attached storage (NAS), opticalstorage, tape drive, backup system, cloud storage, or any combination ofthe foregoing by way of nonlimiting example. Such a medium could alsoinclude instructions programmed into an FPGA, or encoded in hardware onan ASIC or processor.

FIG. 3 is a block diagram of a software-defined network 300, accordingto one or more examples of the present specification. In softwaredefined networking (SDN), a single configuration utility (often agraphical interface or browser interface) may be used to manage networkresources at a high level, with very little manual human interventioninto the details of the network. SDN may provide a data plane that isseparate from a control plane, to separate management functions fromdata functions. Another benefit of SDNs is that they may be based onopen standards, thus providing portability between systems, andalleviating issues of vendor lock-in.

SDN 300 is controlled by an SDN controller 310, which may include, forexample, a VM controller 312, a storage controller 314, and a networkcontroller 316. Other SDN controller functions may also be provided inother embodiments, and not every embodiments is required to have theforegoing elements. SDN controller 310 provides an orchestration layer320. The orchestration layer may employ an open orchestration protocol,such as the OpenStack cloud operating system.

Orchestration layer 320 may include various plugin components that canbe used as interfaces to control data center resources. These pluginsmay interact with orchestration layer 320 via a set of standardized andopen APIs, thus enabling different vendors to provide different plugins.In many cases, data center resources can all be managed via a singlegraphical interface provided by orchestration layer 320. For example,OpenStack currently provides a dashboard called “Horizon,” whichprovides a monolithic interface that enables an administrator to fullyconfigure and administer a data center.

In some embodiments, orchestration layer 320 provides a northbound API,on which may be instantiated various applications or functions, such asapplications 324-1 - 324-N illustrated here. Each application 324 may beprovided on a discrete virtual network device or container (referred toherein generically as “network devices”). Applications 324 may beconfigured to perform a variety of network functions, such as, by way ofnonlimiting example, load-balancing, firewall, deep packet inspection(DPI), DNS, antivirus, or any other suitable network function. Theparticular arrangement of interconnections between network devices 330and from network devices 330 to host devices 340 may be determined bythe particular network configuration and needs. Thus, the specificconfiguration of FIG. 3 should be understood to be an illustrativeexample only.

Orchestration layer may provide communication protocols, including insome embodiments OpenFlow. OpenFlow centralizes networking intelligenceinto an SDN controller such as SDN controller 310. Using OpenFlow,switches and routers do not need to use prior data exchange protocols tolearn about the network topology and other environmental factors. Thesetopology data are replicated across all switches and routers in thenetwork, and each one maintains forwarding tables. In contrast, anOpenFlow controller (which may be, for example, a function of networkcontroller 316) provides centralized network management that maintainsthe network state in a central repository. Network controller 316 canthen update forwarding tables as the network topology evolves orchanges, generate a forwarding table for each switch or router, and pushthem out to network devices as necessary. This realizes separation ofthe control plane and the data plane. Individual network devices mayimplement the OpenFlow API by adding an OpenFlow shim layer thattranslates OpenFlow API function calls into native function calls forthe switch or router.

Note that while this illustration shows a single SDN controller 310, inlarge data centers, an orchestration controller may control a number ofdistributed SDN controllers. In some embodiments, SDN controllers andorchestrators can communicate with one another to remain synchronizationand state information via a sideband, such as a separate, lower-speedEthernet connection.

FIG. 4 is a block diagram of a network function virtualization (NFV)infrastructure 400 according to one or more examples of the presentspecification. NFV is an aspect of network virtualization that isgenerally considered distinct from, but that can still interoperate withSDN. For example, virtual network functions (VNFs) may operate withinthe data plane of an SDN deployment. NFV was originally envisioned as amethod for providing reduced capital expenditure (Capex) and operatingexpenses (Opex) for telecommunication services. One feature of NFV isreplacing proprietary, special-purpose hardware appliances with virtualappliances running on commercial off-the-shelf (COTS) hardware within avirtualized environment. In addition to Capex and Opex savings, NFVprovides a more agile and adaptable network. As network loads change,virtual network functions (VNFs) can be provisioned (“spun up”) orremoved (“spun down”) to meet network demands. For example, in times ofhigh load, more load balancer VNFs may be spun up to distribute trafficto more workload servers (which may themselves be virtual machines). Intimes when more suspicious traffic is experienced, additional firewallsor deep packet inspection (DPI) appliances may be needed.

Because NFV started out as a telecommunications feature, many NFVinstances are focused on telecommunications. However, NFV is not limitedto telecommunication services. In a broad sense, NFV includes one ormore VNFs running within a network function virtualizationinfrastructure (NFVI), such as NFVI 400. Often, the VNFs are inlineservice functions that are separate from workload servers or othernodes. These VNFs can be chained together into a service chain, whichmay be defined by a virtual subnetwork, and which may include a serialstring of network services that provide behind-the-scenes work, such assecurity, logging, billing, and similar.

Like SDN, NFV is a subset of network virtualization. Thus, the networkas illustrated in FIG. 4 may be defined instead of or in addition to thenetwork of FIG. 3 . In other words, certain portions of the network mayrely on SDN, while other portions (or the same portions) may rely onNFV.

In the example of FIG. 4 , an NFV orchestrator 401 manages a number ofthe VNFs 412 running on an NFVI 400. NFV requires nontrivial resourcemanagement, such as allocating a very large pool of compute resourcesamong appropriate numbers of instances of each VNF, managing connectionsbetween VNFs, determining how many instances of each VNF to allocate,and managing memory, storage, and network connections. This may requirecomplex software management, thus making NFV orchestrator 401 a valuablesystem resource. Note that NFV orchestrator 401 may provide abrowser-based or graphical configuration interface, and in someembodiments may be integrated with SDN orchestration functions.

Note that NFV orchestrator 401 itself may be virtualized (rather than aspecial-purpose hardware appliance). NFV orchestrator 401 may beintegrated within an existing SDN system, wherein an operations supportsystem (OSS) manages the SDN. This may interact with cloud resourcemanagement systems (e.g., OpenStack) to provide NFV orchestration. AnNFVI 400 may include the hardware, software, and other infrastructure toenable VNFs to run. This may include a hardware platform 402 on whichone or more VMs 404 may run. For example, hardware platform 402-1 inthis example runs VMs 404-1 and 404-2. Hardware platform 402-2 runs VMs404-3 and 404-4. Each hardware platform may include a hypervisor 420,virtual machine manager (VMM), or similar function, which may includeand run on a native (bare metal) operating system, which may be minimalso as to consume very few resources.

Hardware platforms 402 may be or comprise a rack or several racks ofblade or slot servers (including, e.g., processors, memory, andstorage), one or more data centers, other hardware resources distributedacross one or more geographic locations, hardware switches, or networkinterfaces. An NFVI 400 may also include the software architecture thatenables hypervisors to run and be managed by NFV orchestrator 401.

Running on NFVI 400 are a number of VMs 404, each of which in thisexample is a VNF providing a virtual service appliance. Each VM 404 inthis example includes an instance of the Data Plane Development Kit(DVDK), a virtual operating system 408, and an application providing theVNF 412.

Virtualized network functions could include, as nonlimiting andillustrative examples, firewalls, intrusion detection systems, loadbalancers, routers, session border controllers, deep packet inspection(DPI) services, network address translation (NAT) modules, or callsecurity association.

The illustration of FIG. 4 shows that a number of VNFs 404 have beenprovisioned and exist within NFVI 400. This figure does not necessarilyillustrate any relationship between the VNFs and the larger network, orthe packet flows that NFVI 400 may employ.

The illustrated DPDK instances 416 provide a set of highly-optimizedlibraries for communicating across a virtual switch (vSwitch) 422. LikeVMs 404, vSwitch 422 is provisioned and allocated by a hypervisor 420.The hypervisor uses a network interface to connect the hardware platformto the data center fabric (e.g., an HFI). This HFI may be shared by allVMs 404 running on a hardware platform 402. Thus, a vSwitch may beallocated to switch traffic between VMs 404. The vSwitch may be a puresoftware vSwitch (e.g., a shared memory vSwitch), which may be optimizedso that data are not moved between memory locations, but rather, thedata may stay in one place, and pointers may be passed between VMs 404to simulate data moving between ingress and egress ports of the vSwitch.The vSwitch may also include a hardware driver (e.g., a hardware networkinterface IP block that switches traffic, but that connects to virtualports rather than physical ports). In this illustration, a distributedvSwitch 422 is illustrated, wherein vSwitch 422 is shared between two ormore physical hardware platforms 402.

FIG. 5 is a block diagram of a virtualized network functioninfrastructure (VNFI), according to one or more examples of the presentspecification. Note that the illustration of FIG. 5 may in someembodiments illustrate a more detailed view of certain aspects of FIG. 4. In this case, a host hardware platform includes a host kernel space502, a memory management unit (MMU) 520, an IOMMU 516, and host physicalmemory 524. vHost provisions a VM 544 that will map its guest physicalmemory (GPM) to host physical memory (HPM).

The embodiment of FIG. 5 illustrates a software-only vSwitch, and isprovided here to illustrate that the hardware assisted vSwitch of thepresent specification can be migrated to systems that employsoftware-only vSwitches.

In this embodiment, an open vSwitch (OVS) 504 is provided. OVS is anopen source implementation of vSwitch for hardware virtualizationenvironments, which supports multiple protocols and standards used incomputer networks. This embodiment illustrates an example where the OVSis provided in software.

Host kernel space 502 also includes vHost-net 508, which may providehost virtual address (HVA) to HPA mapping via MMU 520. Finally, hostkernel space 502 includes kernel virtual machine (KVM) 512 whichprovides standard KVM services, such as are provided in the standardLinux KVM. As discussed above, a virtual ring (vring) direct memoryaccess (DMA) mechanism 526 may be provided. Signals such as kicks, TX,and RX may be provided by vHost-net 508 through MMU 520 via vring 526 toVM 544, which maps from memory access operations to guest pages 530within guest 528. Guest 528 may also include its own virtio-net driver534. VM 544 also includes a memory application program interface (MAPI)which communicates with vHost-net 508 to provide GPA mapping for guest528. VM 544 also maintains guest visible device states 538 which may besynchronized with a logging function of host kernel space 502 via KVM512.

FIG. 6 illustrates a VNFI, according to one or more examples of thepresent specification. In the example of FIG. 6 , many aspects aresubstantially similar to the software-only vSwitch of FIG. 5 . However,FIG. 6 provides a hardware assisted vSwitch with vHost offload into thehardware.

Specifically, this VNFI includes a hardware (HW) vSwitch 604 withexact-match cache (EMC) capability. HW vSwitch 604 may be a hardware IPblock, for example, a circuit similar or identical to the controlhardware in a hardware switch or network interface device. In existingsystems, VM 644 would require a specific driver to operate with HWvSwitch 604. But in this case, vHost-net hardware 610 provides the“driver,” abstracting out any functionality specific to HW vSwitch 604,so that it can interact with guest 620 via standardized virtio-netdriver 634. HW vSwitch 604 determines how to handle incoming packets,and is also provided in hardware in this example.

Similar to FIG. 5 , FIG. 6 includes a host kernel space 602, a hostmemory 624, an MMU 620, an IOMMU 616, and a VM 644. As before, guestpages 630 of guest 628 are mapped to addresses in host memory 624. Avring DMA 626 is included that provides signals such as kick, TX, andRX.

Also in this embodiment, host kernel space 602 includes an IOMMU driver606 that communicatively couples to IOMMU 616.

As before, vHost 610 provides mapping to guest pages 630, but in thiscase HPA mapping is provided via IOMMU 616. Furthermore, signals fromvHost-net hardware 610 to guest 628 are provided via IOMMU 616.

VM 644 may be substantially similar or identical to VM 544. It includesa virtio-net driver 634, guest pages 630, guest visible device states638, and a memory API 640. Because VM 644 can be the same for either asoftware-only vSwitch or a hardware assisted vSwitch with vHostoffloaded, it is possible to provide live migration betweensoftware-only vSwitches and hardware assisted vSwitches. For example,guest 628 may be started up with virtio-net driver 634, with vHost-nethardware 610 on the host side. Note that in some cases this may requirememory pinning as described above, at least for the virtio-net memoryportion. This version of vHost-net 610 is hardware offloaded. Thus,instead of starting up a new vHost-net thread to handle the queues,vHost-net may assign a VF to this guest and pass the memory location forwhere the vrings for the guest should be located. Guest 628 may copydata into its guest private memory and put a descriptor pointing to itinto the descriptor table. The writing of this memory (if it is thefirst time this memory has been used) may trigger a page fault in thehypervisor. The hypervisor may then set up the guest memory and send anassociation between GPA and HVA to vHost-net hardware 610.

vHost-net hardware 610 now maps the GPA into the HPA using the mappingfrom QEMU. Normally, this information may be provided to the threadhandling the RX or TX for that guest, but in this embodiment vHost-nethardware 610 instead is modified to program this into IOMMU 616. Guest628 may then update the avail ring via an MMIO write, and the act ofwriting the avail index kicks the backend with the new data. The hosthardware platform then receives the kick and reads the avail index. Itcan then read the host physical memory using the GPA provided in thedescriptor queue. The IOMMU maps this GPA into an actual HPA location ofmemory, thus allowing the memory operation to correctly complete. Thehost hardware platform can pull all metadata and data out of memory andpush it into the hardware vSwitch pipeline.

When the host hardware platform completes reading out the packet, it mayoptionally fire an interrupt indicating that it is done. vHost-nethardware 610 may be modified to watch the hardware interrupt andtranslate it into an IRQFD back to guest 628.

FIG. 7 is a block diagram of VM migration 700, according to one or moreexamples of the present specification. This diagram illustrates that asingle VM 704 can be migrated between different hardware platforms 712as illustrated in the preceding two FIGURES.

For example, VM 704 may start on platform 1 712-1. Platform 1 712-1includes a vSwitch with hardware species 1 720. vSwitch 720 may be, forexample, a hardware assisted vSwitch with the hardware vSwitch functionsprovided by a switching chip from vendor A. This switch may notnecessarily be compatible with a switching chip provided by vendor B.

However, because of the vHost offload hardware illustrated in theprevious examples, VM 704 can be seamlessly migrated to platform 2712-2. Platform 2 712-2 may be, for example, a known platform thatprovides a software-only vSwitch 722. Next, VM 704 may be seamlesslymigrated to platform 3 712-3. Platform 3 712-3 may include a secondhardware assisted vSwitch 724 with vSwitch hardware provided by vendorB, which may be incompatible with hardware provided by vendor A.However, because of the vHost-net hardware abstraction provided in theprevious examples, this live migration is possible. Finally, VM 704could be migrated back to platform 1 712-1 without the need forreconfiguration. This illustration is provided as a nonlimiting exampleonly, and it should be understood that other types of configurations arecompatible with the teachings of this specification.

FIG. 8 is a signal flow diagram illustrating a guest transmit operation,according to one or more examples of the present specification. In thiscase, guest 804 (which includes a virtio-net driver) operates under ahypervisor 808, which runs on a hardware platform including hardware816, and vHost-net hardware 812.

In operation 1, guest 804 is started with a virtio-net driver thatprovides an abstraction layer for a vHost-net hardware 812 on thehardware platform. Note that vHost-net 812 is a hardware offloadedvHost. Also note that instead of starting up a new vHost-net thread tohandle the queues, vHost-net 812 assigns a VF to guest 804 and passesthe memory location where the vrings for the guest should be located.Guest 804 then copies data into its guest private memory, and puts adescriptor pointing to it into the descriptor table.

In one example, this writing of the memory (if it is the first time thismemory has been used) triggers a page fault on hypervisor 808. Inoperation 3, responsive to the page fault, the hypervisor may set up theguest memory.

In operation four, hypervisor 808 sets up an association between GPA andHVA that is provided to vHost-net 812.

In operation 5, vHost-net 812 maps this GPA into HPA using the mappingfrom QEMU. In existing systems, this information could be given to thethread handling the RX or TX for that request. However, in this case,vHost-net is modified to program this into an IOMMU.

In operation 6, guest 804 updates the avail ring via an MMIO write, andthe act of writing the avail index may kick the backend with new data.

The hardware device gets the kick and reads the avail index. It thengoes to read the host physical memory using the GPA provided in thedescriptor queue. The IOMMU may remap the GPA into an HPA location inmemory, thus allowing the memory operation to complete correctly. Thehardware platform can now pull all the metadata and data out of memoryand push it into the hardware vSwitch pipeline.

In operation 7, when the hardware device is finished reading out thepacket, it may optionally fire an interrupt to vHost-net 812, indicatingthat it is done. In operation 8, vHost-net 812 may watch the hardwareinterrupt, and translate it into an IRQFD back to guest 804.

FIG. 9 is a signal flow diagram illustrating a live migration event,according to one or more examples of the present specification. During alive migration event, the set of active mappings in the IOMMU may besent back to QEMU. This allows QEMU to dirty all active pages, to getthe complete set of state and RAM for the migration event. The currentactive state of the RX or TX queue is logged to be sent alongside theVM, thus ensuring that the device state is reinstated at thedestination. This works the same as in the previous examples, with theexception that kicks from the guest may be done through filedescriptors. The flow for transmit works similarly, with the exceptionthat the guest does not kick the hardware device directly, but insteadthis may be relayed via vHost-net or another vHost-net modification.Note that some examples may require the hardware to provide a secondarymemory space for vHost-net to do the kick if the page in memory has tobe used both for kicks and for the avail index from the guests.

During a live migration event, the software pulls the following out ofhardware. First, it pulls IOMMU. The current set of pages being mappedinto the IOMMU may be pulled out so that they can be dirtied. When thesepages are accessed on the destination machine, a page fault will triggerallowing the destination system’s state to be repopulated (either inhardware or in software, depending on its implementation). It may alsopull the vring state out of hardware. In both hardware and softwarevring implementations, the device state should be carried with the VM tothe destination. The hardware queues may provide exactly the sameinformation about its state as a vHost-net thread in a software-onlyvSwitch. This includes the current state of the descriptor tables,indices, and interrupt state. At the destination, the vring state may bepopulated with this information and warm started with the currentrunning state.

During a live migration event, additional state and/or configuration maybe managed in the vSwitch. This may be a separate operation from what ishandled in the host interface.

As shown in FIG. 9 , at operation 1, during a live migration old host904 initiates a live migration event on a particular VM. It sends amessage to vHost-net 908 telling it to stop fetching new work.

vHost-net 908 then tells the underlying hardware to stop fetching newwork.

In operation 2, on old host 904, the guest completes its work inprogress. New transmit packets start to pile up in the guest memory, andincoming received packets are dropped.

In operation 3, old host 904 signals to vHost-net 908 that the stall iscompleted. Old host 904 then freezes the guest and copies the guest RAM.

In operation 4, the host reads configuration state and statisticspopulated in the device as well as the mappings programmed into theIOMMU.

In operation 5, this information is carried alongside the VM to new host912.

In operation 6, new host 912 creates the new VM, then copies the RAM andthe additional state information. The vHost on new host 912 is given therunning state, to start it up in the same state as the source. New host912 vHost-net sends the configuration state and statistics to the vrings(either in hardware or software) to warm start them in the same state asat the source.

In operation 7, the new host is unfrozen, and an event is sent tovHost-net to restart. The new vHost-net enables the vrings, either insoftware or in hardware.

FIG. 10 illustrates a VNFI, according to one or more examples of thepresent specification. FIG. 10 is similar to FIG. 6 in many respects,and further illustrates interrupt remapping via the IOMMU and doorbellmapping via the MMU EPT.

This VNFI includes a hardware (HW) vSwitch 1004 with exact-match cache(EMC) capability. HW vSwitch 1004 may be a hardware IP block, forexample, a circuit similar or identical to the control hardware in ahardware switch or network interface device. In existing systems, VM1044 would require a specific driver to operate with HW vSwitch 1004.But in this case, vHost-net hardware 1010 provides the “driver,”abstracting out any functionality specific to HW vSwitch 1004, so thatit can interact with guest 1020 via standardized virtio-net driver 1034.HW vSwitch 1004 determines how to handle incoming packets, and is alsoprovided in hardware in this example.

Similar to FIG. 5 , FIG. 6 includes a host kernel space 1002, a hostmemory 1024, an MMU 1020, an IOMMU 1016, and a VM 1044. As before, guestpages 1030 of guest 1028 are mapped to addresses in host memory 1024. Avring DMA 1026 is included that provides signals such as kick, TX, andRX.

Also in this embodiment, host kernel space 1002 includes an IOMMU driver1006 that communicatively couples to IOMMU 1016.

As before, vHost 1010 provides mapping to guest pages 1030, but in thiscase HPA mapping is provided via IOMMU 1016. Furthermore, signals fromvHost-net hardware 1010 to guest 1028 are provided via IOMMU 1016.

VM 1044 may be substantially similar or identical to VM 544. It includesa virtio-net driver 1034, guest pages 1030, guest visible device states1038, and a memory API 1040. Because VM 1044 can be the same for eithera software-only vSwitch or a hardware assisted vSwitch with vHostoffloaded, it is possible to provide live migration betweensoftware-only vSwitches and hardware assisted vSwitches. For example,guest 1028 may be started up with virtio-net driver 1034, with vHost-nethardware 1010 on the host side. Note that in some cases this may requirememory pinning as described above, at least for the virtio-net memoryportion. This version of vHost-net 1010 is hardware offloaded. Thus,instead of starting up a new vHost-net thread to handle the queues,vHost-net may assign a VF to this guest and pass the memory location forwhere the vrings for the guest should be located. Guest 1028 may copydata into its guest private memory and put a descriptor pointing to itinto the descriptor table. The writing of this memory (if it is thefirst time this memory has been used) may trigger a page fault in thehypervisor. The hypervisor may then set up the guest memory and send anassociation between GPA and HVA to vHost-net hardware 1010.

vHost-net hardware 1010 now maps the GPA into the HPA using the mappingfrom QEMU. Normally, this information may be provided to the threadhandling the RX or TX for that guest, but in this embodiment vHost-nethardware 1010 instead is modified to program this into IOMMU 1016. Guest1028 may then update the avail ring via an MMIO write, and the act ofwriting the avail index kicks the backend with the new data. The hosthardware platform then receives the kick and reads the avail index. Itcan then read the host physical memory using the GPA provided in thedescriptor queue. The IOMMU maps this GPA into an actual HPA location ofmemory, thus allowing the memory operation to correctly complete. Thehost hardware platform can pull all metadata and data out of memory andpush it into the hardware vSwitch pipeline.

When the host hardware platform completes reading out the packet, it mayoptionally fire an interrupt indicating that it is done. vHost-nethardware 1010 may be modified to watch the hardware interrupt andtranslate it into an IRQFD back to guest 1028.

By way of instructions provided in software, an interrupt remappingtable may be provided in IOMMU 1016, enabling vHost offload hardware1010 to pass interrupts into a virtualized guest. By leveraging theinterrupt remapping feature of IOMMU 1016 (e.g., posted interrupt), ahypervisor need not intervene in the relay

Note that in the foregoing examples, in the guest, both the addresstranslation (guest-to-host) and the interrupts are still relayed in thehost. These may represent primary differences in performance betweenvHost offload and VF passthrough. It is possible to make updates purelyon the host side (updates to QEMU, vHost-net, or others) to allow theseto be offloaded completely, thus to close the performance gap. At thatpoint, there would be no need to continue supporting virtio-net VFpassthrough. Thus, embodiments of the present specification anticipatesystems that do not support virtio-net VF passthrough.

Also note that guests that run a poll mode driver such as DPDK may notneed these particular enhancements, because DPDK maps memory inHugePages to avoid the overhead of dynamic pages. DPDK also polls fordata, thus avoiding the overhead of interrupts.

The foregoing outlines features of one or more embodiments of thesubject matter disclosed herein. These embodiments are provided toenable a person having ordinary skill in the art (PHOSITA) to betterunderstand various aspects of the present disclosure. Certainwell-understood terms, as well as underlying technologies and/orstandards may be referenced without being described in detail. It isanticipated that the PHOSITA will possess or have access to backgroundknowledge or information in those technologies and standards sufficientto practice the teachings of the present specification.

The PHOSITA will appreciate that they may readily use the presentdisclosure as a basis for designing or modifying other processes,structures, or variations for carrying out the same purposes and/orachieving the same advantages of the embodiments introduced herein. ThePHOSITA will also recognize that such equivalent constructions do notdepart from the spirit and scope of the present disclosure, and thatthey may make various changes, substitutions, and alterations hereinwithout departing from the spirit and scope of the present disclosure.

In the foregoing description, certain aspects of some or all embodimentsare described in greater detail than is strictly necessary forpracticing the appended claims. These details are provided by way ofnon-limiting example only, for the purpose of providing context andillustration of the disclosed embodiments. Such details should not beunderstood to be required, and should not be “read into” the claims aslimitations. The phrase may refer to “an embodiment” or “embodiments.”These phrases, and any other references to embodiments, should beunderstood broadly to refer to any combination of one or moreembodiments. Furthermore, the several features disclosed in a particular“embodiment” could just as well be spread across multiple embodiments.For example, if features 1 and 2 are disclosed in “an embodiment,”embodiment A may have feature 1 but lack feature 2, while embodiment Bmay have feature 2 but lack feature 1.

This specification may provide illustrations in a block diagram format,wherein certain features are disclosed in separate blocks. These shouldbe understood broadly to disclose how various features interoperate, butare not intended to imply that those features must necessarily beembodied in separate hardware or software. Furthermore, where a singleblock discloses more than one feature in the same block, those featuresneed not necessarily be embodied in the same hardware and/or software.For example, a computer “memory” could in some circumstances bedistributed or mapped between multiple levels of cache or local memory,main memory, battery-backed volatile memory, and various forms ofpersistent memory such as a hard disk, storage server, optical disk,tape drive, or similar. In certain embodiments, some of the componentsmay be omitted or consolidated. In a general sense, the arrangementsdepicted in the FIGURES may be more logical in their representations,whereas a physical architecture may include various permutations,combinations, and/or hybrids of these elements. Countless possibledesign configurations can be used to achieve the operational objectivesoutlined herein. Accordingly, the associated infrastructure has a myriadof substitute arrangements, design choices, device possibilities,hardware configurations, software implementations, and equipmentoptions.

References may be made herein to a computer-readable medium, which maybe a tangible and non-transitory computer-readable medium. As used inthis specification and throughout the claims, a “computer-readablemedium” should be understood to include one or more computer-readablemediums of the same or different types. A computer-readable medium mayinclude, by way of non-limiting example, an optical drive (e.g.,CD/DVD/Blu-Ray), a hard drive, a solid-state drive, a flash memory, orother non-volatile medium. A computer-readable medium could also includea medium such as a read-only memory (ROM), an FPGA or ASIC configured tocarry out the desired instructions, stored instructions for programmingan FPGA or ASIC to carry out the desired instructions, an intellectualproperty (IP) block that can be integrated in hardware into othercircuits, or instructions encoded directly into hardware or microcode ona processor such as a microprocessor, digital signal processor (DSP),microcontroller, or in any other suitable component, device, element, orobject where appropriate and based on particular needs. A nontransitorystorage medium herein is expressly intended to include any nontransitoryspecial-purpose or programmable hardware configured to provide thedisclosed operations, or to cause a processor to perform the disclosedoperations.

Various elements may be “communicatively,” “electrically,”“mechanically,” or otherwise “coupled” to one another throughout thisspecification and the claims. Such coupling may be a direct,point-to-point coupling, or may include intermediary devices. Forexample, two devices may be communicatively coupled to one another via acontroller that facilitates the communication. Devices may beelectrically coupled to one another via intermediary devices such assignal boosters, voltage dividers, or buffers. Mechanically-coupleddevices may be indirectly mechanically coupled.

Any “module” or “engine” disclosed herein may refer to or includesoftware, a software stack, a combination of hardware, firmware, and/orsoftware, a circuit configured to carry out the function of the engineor module, a co-processor, FPGA, ASIC, or any computer-readable mediumas disclosed above. Such modules or engines may, in appropriatecircumstances, be provided on or in conjunction with a hardwareplatform, which may include hardware compute resources such as aprocessor, memory, storage, interconnects, networks and networkinterfaces, accelerators, or other suitable hardware. Such a hardwareplatform may be provided as a single monolithic device (e.g., in a PCform factor), or with some or part of the function being distributed(e.g., a “composite node” in a high-end data center, where compute,memory, storage, and other resources may be dynamically allocated andneed not be local to one another).

There may be disclosed herein flow charts, signal flow diagram, or otherillustrations showing operations being performed in a particular order.Unless otherwise expressly noted, or unless required in a particularcontext, the order should be understood to be a non-limiting exampleonly. Furthermore, in cases where one operation is shown to followanother, other intervening operations may also occur, which may berelated or unrelated. Some operations may also be performedsimultaneously or in parallel. In cases where an operation is said to be“based on” or “according to” another item or operation, this should beunderstood to imply that the operation is based at least partly on oraccording at least partly to the other item or operation. This shouldnot be construed to imply that the operation is based solely orexclusively on, or solely or exclusively according to the item oroperation.

All or part of any hardware element disclosed herein may readily beprovided in a system-on-a-chip (SoC), including a central processingunit (CPU) package. An SoC represents an integrated circuit (IC) thatintegrates components of a computer or other electronic system into asingle chip. Thus, for example, client devices or server devices may beprovided, in whole or in part, in an SoC. The SoC may contain digital,analog, mixed-signal, and radio frequency functions, all of which may beprovided on a single chip substrate. Other embodiments may include amultichip module (MCM), with a plurality of chips located within asingle electronic package and configured to interact closely with eachother through the electronic package.

In a general sense, any suitably-configured circuit or processor canexecute any type of instructions associated with the data to achieve theoperations detailed herein. Any processor disclosed herein couldtransform an element or an article (for example, data) from one state orthing to another state or thing. Furthermore, the information beingtracked, sent, received, or stored in a processor could be provided inany database, register, table, cache, queue, control list, or storagestructure, based on particular needs and implementations, all of whichcould be referenced in any suitable timeframe. Any of the memory orstorage elements disclosed herein, should be construed as beingencompassed within the broad terms “memory” and “storage,” asappropriate.

Computer program logic implementing all or part of the functionalitydescribed herein is embodied in various forms, including, but in no waylimited to, a source code form, a computer executable form, machineinstructions or microcode, programmable hardware, and variousintermediate forms (for example, forms generated by an assembler,compiler, linker, or locator). In an example, source code includes aseries of computer program instructions implemented in variousprogramming languages, such as an object code, an assembly language, ora high-level language such as OpenCL, FORTRAN, C, C++, JAVA, or HTML foruse with various operating systems or operating environments, or inhardware description languages such as Spice, Verilog, and VHDL. Thesource code may define and use various data structures and communicationmessages. The source code may be in a computer executable form (e.g.,via an interpreter), or the source code may be converted (e.g., via atranslator, assembler, or compiler) into a computer executable form, orconverted to an intermediate form such as byte code. Where appropriate,any of the foregoing may be used to build or describe appropriatediscrete or integrated circuits, whether sequential, combinatorial,state machines, or otherwise.

In one example embodiment, any number of electrical circuits of theFIGURES may be implemented on a board of an associated electronicdevice. The board can be a general circuit board that can hold variouscomponents of the internal electronic system of the electronic deviceand, further, provide connectors for other peripherals. Any suitableprocessor and memory can be suitably coupled to the board based onparticular configuration needs, processing demands, and computingdesigns. Note that with the numerous examples provided herein,interaction may be described in terms of two, three, four, or moreelectrical components. However, this has been done for purposes ofclarity and example only. It should be appreciated that the system canbe consolidated or reconfigured in any suitable manner. Along similardesign alternatives, any of the illustrated components, modules, andelements of the FIGURES may be combined in various possibleconfigurations, all of which are within the broad scope of thisspecification.

Numerous other changes, substitutions, variations, alterations, andmodifications may be ascertained to one skilled in the art and it isintended that the present disclosure encompass all such changes,substitutions, variations, alterations, and modifications as fallingwithin the scope of the appended claims. In order to assist the UnitedStates Patent and Trademark Office (USPTO) and, additionally, anyreaders of any patent issued on this application in interpreting theclaims appended hereto, Applicant wishes to note that the Applicant: (a)does not intend any of the appended claims to invoke paragraph six (6)of 35 U.S.C. section 112 (pre-AIA) or paragraph (f) of the same section(post-AIA), as it exists on the date of the filing hereof unless thewords “means for” or “steps for” are specifically used in the particularclaims; and (b) does not intend, by any statement in the specification,to limit this disclosure in any way that is not otherwise expresslyreflected in the appended claims.

Example Implementations

The following examples are provided by way of illustration.

Example 1 includes a computing apparatus for providing ahardware-assisted virtual switch on a host, comprising: a hardwarevirtual switch (vSwitch) circuit; and a hardware virtual host (vHost)circuit, the vHost circuit having an interface driver specific to thehardware vSwitch and configured to provide a vHost data plane to:provide a plurality of hardware queues to communicatively couple thehardware vSwitch to a guest virtual function (VF); and present to avirtual network driver of the guest VF an interface that is backwardcompatible with a software network interface.

Example 2 includes the computing apparatus of example 1, wherein thevirtual network driver is to provide vHost control plane functionality.

Example 3 includes the computing apparatus of example 1, wherein thevHost control plane functionality comprises capability negotiation.

Example 4 includes the computing apparatus of example 1, wherein thevHost control plane functionality comprises live migration.

Example 5 includes the computing apparatus of example 1, wherein thevHost circuit is to receive an eventfd kick and translate the kick intoa hardware write on the host.

Example 6 includes the computing apparatus of example 1, wherein thevHost circuit is to provide an interrupt request (IRQ) relay to receivea host-side hardware IRQ, and translate the host-side IRQ into asoftware IRQ on the guest.

Example 7 includes the computing apparatus of example 1, wherein thevHost circuit is to provide a host-side interrupt request (IRQ) directlyto the VM, comprising translating the host-side IRQ into aguest-specific interrupt via an input/output memory management unit(IOMMU).

Example 8 includes the computing apparatus of example 1 furthercomprising an input/output memory management unit (IOMMU), wherein thevHost circuit is to map a guest physical address (GPA) space into a hostphysical address (HPA) space in the IOMMU, wherein host hardware is tolocate HPAs via GPAs.

Example 9 includes the computing apparatus of example 8, wherein thevHost circuit is provide hardware separation of input/output operationsfor a plurality of guest virtual machines.

Example 10 includes the computing apparatus of example 8, wherein thevHost circuit is to receive a kick, read an “avail” index, and read froma host physical memory address according to a GPA in a descriptor queueof the avail index.

Example 11 includes the computing apparatus of example 8, wherein thevHost circuit is to pin GPA pages into host memory.

Example 12 includes the computing apparatus of example 11, whereinpinning GPA pages into host memory comprises pinning only memory for thevirtual network driver.

Example 13 includes the computing apparatus of example 1, wherein thevHost circuit is to receive a live migration event notification, andresponsive to the live migration event notification, dirty active memorypages of the guest VF and swap active memory pages out of cache.

Example 14 includes the computing apparatus of example 1, wherein thevHost circuit is to perform pure host-side updates of guest-to-hostaddress translation and interrupts, wherein VF passthrough for thevirtual network driver is not provided to the guest.

Example 15 includes the computing apparatus of any of examples 1 - 14,wherein the hardware vSwitch comprises a hardware IP block.

Example 16 includes one or more tangible, non-transitorycomputer-operable mediums having encoded thereon instructions to providea hardware virtual host (vHost) to: communicatively couple to a hardwarevirtual switch (vSwitch) circuit; provide an interface driver specificto the hardware vSwitch and configured to provide a vHost data plane to:provide a plurality of hardware queues to communicatively couple thehardware vSwitch to a guest virtual function (VF); and present to avirtual network driver of the guest VF an interface that is backwardcompatible with a software network interface.

Example 17 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the virtual networkdriver is to provide vHost control plane functionality.

Example 18 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the vHost control planefunctionality comprises capability negotiation.

Example 19 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the vHost control planefunctionality comprises live migration.

Example 20 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the hardware vHost isto receive an eventfd kick and translate the kick into a hardware writeon the host.

Example 21 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the hardware vHost isto provide an interrupt request (IRQ) relay to receive a host-sidehardware IRQ, and translate the host-side IRQ into a software IRQ on theguest.

Example 22 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the hardware vHost isto communicatively couple to an input/output memory management unit(IOMMU), wherein the hardware vHost is to map a guest physical address(GPA) space into a host physical address (HPA) space in the IOMMU,wherein host hardware is enabled to locate HPAs via GPAs.

Example 23 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the hardware vHost isto receive a kick, read an “avail” index, and read from a host physicalmemory address according to a GPA in a descriptor queue of the availindex.

Example 24 includes the one or more tangible, non-transitorycomputer-operable mediums of example 23, wherein the hardware vHost isto pin GPA pages into host memory.

Example 25 includes the one or more tangible, non-transitorycomputer-operable mediums of example 23, wherein pinning GPA pages intohost memory comprises pinning only memory for the virtual networkdriver.

Example 26 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the hardware vHost isto receive a live migration event notification, and responsive to thelive migration event notification, dirty active memory pages of theguest VF and swap active memory pages out of cache.

Example 27 includes the one or more tangible, non-transitorycomputer-operable mediums of example 16, wherein the hardware vHost isto perform pure host-side updates of guest-to-host address translationand interrupts, wherein VF passthrough for the virtual network driver isnot provided to the guest.

Example 28 includes the one or more tangible, non-transitorycomputer-operable mediums of any of examples 16 - 27, wherein theinstructions comprise an application-specific integrated circuit.

Example 29 includes the one or more tangible, non-transitorycomputer-operable mediums of any of examples 16 - 27, wherein theinstructions comprise instructions or information to manufacture anintegrated circuit.

Example 30 includes the one or more tangible, non-transitorycomputer-operable mediums of any of examples 16 - 27, wherein theinstructions comprise a field-programmable gate array.

Example 31 includes the one or more tangible, non-transitorycomputer-operable mediums of any of examples 16 - 27, wherein theinstructions comprise instructions to program a field-programmable gatearray.

Example 32 includes the one or more tangible, non-transitorycomputer-operable mediums of any of examples 16 - 27, wherein theinstructions comprise an intellectual property block.

Example 33 includes a computer-implemented method of providinghardware-assisted virtual host (vHost) services to a hardware virtualswitch (vSwitch), comprising: communicatively coupling to the hardwarevSwitch; providing an interface driver specific to the hardware vSwitchand configured to provide a vHost data plane: providing a plurality ofhardware queues to communicatively couple the hardware vSwitch to aguest virtual function (VF); and presenting to a virtual network driverof the guest VF an interface that is backward compatible with a softwarenetwork interface.

Example 34 includes the method of example 33, wherein the virtualnetwork driver is to provide vHost control plane functionality.

Example 35 includes the method of example 33, wherein the vHost controlplane functionality comprises capability negotiation.

Example 36 includes the method of example 33, wherein the vHost controlplane functionality comprises live migration.

Example 37 includes the method of example 33, further comprisingreceiving an eventfd kick and translating the kick into a hardware writeon the host.

Example 38 includes the method of example 33, further comprisingproviding an interrupt request (IRQ) relay to receive a host-sidehardware IRQ, and translating the host-side IRQ into a software IRQ onthe guest.

Example 39 includes the method of example 33, further comprisingcommunicatively coupling to an input/output memory management unit(IOMMU) and mapping a guest physical address (GPA) space into a hostphysical address (HPA) space in the IOMMU, wherein a host hardware isenabled to locate HPAs via GPAs.

Example 40 includes the method of example 33, further comprisingreceiving a kick, reading an “avail” index, and reading from a hostphysical memory address according to a GPA in a descriptor queue of theavail index.

Example 41 includes the method of example 40, further comprising pinningGPA pages into host memory.

Example 42 includes the method of example 41, wherein pinning GPA pagesinto host memory comprises pinning only memory for the virtual networkdriver.

Example 43 includes the method of example 33, further comprisingreceiving a live migration event notification, and responsive to thelive migration event notification, dirty active memory pages of theguest VF and swap active memory pages out of cache.

Example 44 includes the method of example 33, further comprisingperforming pure host-side updates of guest-to-host address translationand interrupts, wherein VF passthrough for the virtual network driver isnot provided to the guest.

Example 45 includes an apparatus comprising means for performing themethod of any of examples 33 - 44.

Example 46 includes one or more tangible, non-transitorycomputer-operable mediums having encoded thereon instructions forproviding the method of any of examples 33 - 44.

Example 47 includes the one or more tangible, non-transitorycomputer-operable mediums of example 46, wherein the instructionscomprise an application-specific integrated circuit.

Example 48 includes the one or more tangible, non-transitorycomputer-operable mediums of example 46, wherein the instructionscomprise instructions or information to manufacture an integratedcircuit.

Example 49 includes the one or more tangible, non-transitorycomputer-operable mediums of example 46, wherein the instructionscomprise a field-programmable gate array.

Example 50 includes the one or more tangible, non-transitorycomputer-operable mediums of example 46, wherein the instructionscomprise instructions to program a field-programmable gate array.

Example 51 includes the one or more tangible, non-transitorycomputer-operable mediums of example 46, wherein the instructionscomprise an intellectual property block.

1. Non-transitory machine-readable memory storing instructions for beingexecuted by at least one machine, the at least one machine to beassociated with server system hardware, the server system hardware beingfor use in association with network functions, the network functionsbeing associated, at least in part, with virtualized network functions,the server system hardware to be used in association withOpenstack-associated network function virtualization (NFV)orchestration, the server system hardware comprising hardware platforms,the hardware platforms to execute virtual machine instances for use inproviding the virtual network functions, the instructions, when executedby the at least one machine, resulting in the server system hardwarebeing configured for performance of operations comprising: providing atleast one virtualized distributed switch for use in forwarding trafficamong the virtual machine instances, the at least one virtualizeddistributed switch to be distributed in the hardware platforms; anddynamically allocating resources associated with the server systemhardware to at least one network function-related instance for use inproviding at least one of the virtual network functions, the resourcesbeing configurable to be distributed among multiple geographiclocations; wherein: at least one virtualized distributed switch isconfigurable for use with data plane development kit (DPDK) poll modedriver (PMD); the server system hardware is configurable to comprise atleast one accelerator; the at least one of the virtual network functionsis configurable to be associated, at least in part, with atelecommunications network function virtualization infrastructure; theat least one of the virtual network functions is configurable toimplement, at least in part, a session border controller; at least oneother of the virtual network functions is configurable to implement, atleast in part, one or more of: a firewall; a router; network addresstranslation; and/or load balancing; and the Openstack-associated NFVorchestration is to be implemented via at least one container.
 2. Thenon-transitory machine-readable memory of claim 1, wherein: the serversystem hardware is for use with at least one switching and/or routingfabric; and the at least one switching and/or routing fabric isconfigurable for use in network traffic communication (1) to and from atleast one wide area network, and/or (2) in a data center.
 3. Thenon-transitory machine-readable memory of claim 2, wherein: at leastcertain of the virtual network functions are configurable to provide atleast one service chain.
 4. The non-transitory machine-readable memoryof claim 3, wherein the operations also comprise: at least one livemigration operation associated, at least in part, with at least one ofthe virtual machine instances; and executing, by the server systemhardware, at least one hypervisor for use in executing the virtualmachine instances.
 5. A method implemented using server system hardware,the server system hardware being for use in association with networkfunctions, the network functions being associated, at least in part,with virtualized network functions, the server system hardware to beused in association with Openstack-associated network functionvirtualization (NFV) orchestration, the server system hardwarecomprising hardware platforms, the hardware platforms to execute virtualmachine instances for use in providing the virtual network functions,the method comprising: providing at least one virtualized distributedswitch for use in forwarding traffic among the virtual machineinstances, the at least one virtualized distributed switch to bedistributed in the hardware platforms; and dynamically allocatingresources associated with the server system hardware to at least onenetwork function-related instance for use in providing at least one ofthe virtual network functions, the resources being configurable to bedistributed among multiple geographic locations; wherein: at least onevirtualized distributed switch is configurable for use with data planedevelopment kit (DPDK) poll mode driver (PMD); the server systemhardware is configurable to comprise at least one accelerator; the atleast one of the virtual network functions is configurable to beassociated, at least in part, with a telecommunications network functionvirtualization infrastructure; the at least one of the virtual networkfunctions is configurable to implement, at least in part, a sessionborder controller; at least one other of the virtual network functionsis configurable to implement, at least in part, one or more of: afirewall; a router; network address translation; and/or load balancing;and the Openstack-associated NFV orchestration is to be implemented viaat least one container.
 6. The method of claim 5, wherein: the serversystem hardware is for use with at least one switching and/or routingfabric; and the at least one switching and/or routing fabric isconfigurable for use in network traffic communication (1) to and from atleast one wide area network, and/or (2) in a data center.
 7. The methodof claim 6, wherein: at least certain of the virtual network functionsare configurable to provide at least one service chain.
 8. The method ofclaim 7, wherein the method further comprises: executing at least onelive migration operation associated, at least in part, with at least oneof the virtual machine instances; and executing, by the server systemhardware, at least one hypervisor for use in executing the virtualmachine instances.
 9. Server system hardware for use, when the serversystem hardware is in operation, in association with network functions,the network functions being associated, at least in part, withvirtualized network functions, the server system hardware to be used inassociation with Openstack-associated network function virtualization(NFV) orchestration, the server system hardware comprising: hardwareplatforms to execute virtual machine instances for use in providing thevirtual network functions; at least one virtualized distributed switchto be provided, when the server system hardware is in the operation, foruse in forwarding traffic among the virtual machine instances, the atleast one virtualized distributed switch to be distributed in thehardware platforms; and resources to be dynamically allocated to atleast one network function-related instance for use in providing atleast one of the virtual network functions, the resources beingconfigurable to be distributed among multiple geographic locations;wherein: at least one virtualized distributed switch is configurable foruse with data plane development kit (DPDK) poll mode driver (PMD); theserver system hardware is configurable to comprise at least oneaccelerator; the at least one of the virtual network functions isconfigurable to be associated, at least in part, with atelecommunications network function virtualization infrastructure; theat least one of the virtual network functions is configurable toimplement, at least in part, a session border controller; at least oneother of the virtual network functions is configurable to implement, atleast in part, one or more of: a firewall; a router; network addresstranslation; and/or load balancing; and the Openstack-associated NFVorchestration is to be implemented via at least one container.
 10. Theserver system hardware of claim 9, wherein: the server system hardwareis for use with at least one switching and/or routing fabric; and the atleast one switching and/or routing fabric is configurable for use innetwork traffic communication (1) to and from at least one wide areanetwork, and/or (2) in a data center.
 11. The server system hardware ofclaim 10, wherein: at least certain of the virtual network functions areconfigurable to provide at least one service chain.
 12. The serversystem hardware of claim 11, wherein the server system hardware isconfigurable to: execute at least one live migration operationassociated, at least in part, with at least one of the virtual machineinstances; and execute at least one hypervisor for use in executing thevirtual machine instances.
 13. A data center for use in processing data,the data center to be associated with at least one wide area network,the data center comprising: rack servers for use with the at least onewide area network, the rack servers comprising server system hardwarefor use, when the server system hardware is in operation, in associationwith network functions, the network functions being associated, at leastin part, with virtualized network functions, the server system hardwareto be used in association with Openstack-associated network functionvirtualization (NFV) orchestration, the server system hardwarecomprising: hardware platforms to execute virtual machine instances foruse in providing the virtual network functions; at least one virtualizeddistributed switch to be provided, when the server system hardware is inthe operation, for use in forwarding traffic among the virtual machineinstances, the at least one virtualized distributed switch to bedistributed in the hardware platforms; and resources to be dynamicallyallocated to at least one network function-related instance for use inproviding at least one of the virtual network functions, the resourcesbeing configurable to be distributed among multiple geographiclocations; wherein: at least one virtualized distributed switch isconfigurable for use with data plane development kit (DPDK) poll modedriver (PMD); the server system hardware is configurable to comprise atleast one accelerator; the at least one of the virtual network functionsis configurable to be associated, at least in part, with atelecommunications network function virtualization infrastructure; theat least one of the virtual network functions is configurable toimplement, at least in part, a session border controller; at least oneother of the virtual network functions is configurable to implement, atleast in part, one or more of: a firewall; a router; network addresstranslation; and/or load balancing; and the Openstack-associated NFVorchestration is to be implemented via at least one container.
 14. Thedata center of claim 13, wherein: the server system hardware is for usewith at least one switching and/or routing fabric; and the at least oneswitching and/or routing fabric is configurable for use in networktraffic communication (1) to and from at least one wide area network,and/or (2) in a data center.
 15. The data center of claim 14, wherein:at least certain of the virtual network functions are configurable toprovide at least one service chain.
 16. The data center of claim 15,wherein the server system hardware is configurable to: execute at leastone live migration operation associated, at least in part, with at leastone of the virtual machine instances; and execute at least onehypervisor for use in executing the virtual machine instances.